Method of identification from a spatial and spectral object model

ABSTRACT

A method of referencing an imaged object includes, among other things, obtaining a series of images, observing key characteristics of the object in each of the series of images, associating the observed key characteristics with the object; and assigning a unique identifier to the object based upon the associated key characteristics. The series of images includes spectral and spatial imagery. Some of the key characteristics are in the spectral imagery and some of the key characteristics are in the spatial imagery.

BACKGROUND OF THE INVENTION

Hyperspectral cameras are capable of capturing hyperspectral imageframes, or datacubes at video frame rates. These cameras acquire highspatial and spectral resolution imagery. In combination with techniquesrelating to computer vision and spectral analysis, operators ofhyperspectral cameras have engaged in surveillance applications relatingto detection, tracking and identification of imaged objects.

BRIEF DESCRIPTION OF THE INVENTION

One aspect of the invention relates to a method of referencing an imagedobject. The method includes: obtaining a series of images wherein atleast some of the images are spectral and some of the images arespatial; observing key characteristics of the object in each of theseries of images wherein some of the key characteristics are in thespectral images and some of the key characteristics are in the spatialimages; associating the observed key characteristics with the object;and assigning a unique identifier to the object based upon theassociated key characteristics.

Another aspect of the invention relates to a system for referencing animaged object. The system includes: at least one imaging deviceconfigured to record a series of spectral and spatial images of anobject; a processor configured to process the spectral and spatialimages, and software in the processor. The software includesinstructions to: observe key characteristics of the object in each ofthe series of images wherein some of the key characteristics are in thespectral images and some of the key characteristics are in the spatialimages; associate the observed key characteristics with the object; andassign a unique identifier to the object based upon the associated keycharacteristics.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 shows a scenario where a system according to an embodiment of thepresent invention includes two exemplary moving platforms that captureimagery of a vehicle.

FIG. 2 shows a scenario where a system according to an embodiment of thepresent invention includes an exemplary platform that captures imageryof a moving vehicle.

FIG. 3 is a flowchart showing a method of generating a spatial andspectral object model from imagery captured by a system like thatdescribed in FIG. 1.

FIG. 4 demonstrates the spatial portioning of an imaged vehicle used togenerate a spatial and spectral object model.

FIG. 5 is a flowchart showing a method of identification from a spatialand spectral object model.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

In the background and the following description, for the purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of the technology described herein. It will beevident to one skilled in the art, however, that the exemplaryembodiments may be practiced without these specific details. In otherinstances, structures and devices are shown in diagram form in order tofacilitate description of the exemplary embodiments.

The exemplary embodiments are described with reference to the drawings.These drawings illustrate certain details of specific embodiments thatimplement a module, method, or computer program product describedherein. However, the drawings should not be construed as imposing anylimitations that may be present in the drawings. The method and computerprogram product may be provided on any machine-readable media foraccomplishing their operations. The embodiments may be implemented usingan existing computer processor, or by a special purpose computerprocessor incorporated for this or another purpose, or by a hardwiredsystem.

As noted above, embodiments described herein may include a computerprogram product comprising machine-readable media for carrying or havingmachine-executable instructions or data structures stored thereon. Suchmachine-readable media can be any available media, which can be accessedby a general purpose or special purpose computer or other machine with aprocessor. By way of example, such machine-readable media can compriseRAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magneticdisk storage or other magnetic storage devices, or any other medium thatcan be used to carry or store desired program code in the form ofmachine-executable instructions or data structures and that can beaccessed by a general purpose or special purpose computer or othermachine with a processor. When information is transferred or providedover a network or another communication connection (either hardwired,wireless, or a combination of hardwired or wireless) to a machine, themachine properly views the connection as a machine-readable medium.Thus, any such a connection is properly termed a machine-readablemedium. Combinations of the above are also included within the scope ofmachine-readable media. Machine-executable instructions comprise, forexample, instructions and data, which cause a general purpose computer,special purpose computer, or special purpose processing machines toperform a certain function or group of functions.

Embodiments will be described in the general context of method stepsthat may be implemented in one embodiment by a program product includingmachine-executable instructions, such as program codes, for example, inthe form of program modules executed by machines in networkedenvironments. Generally, program modules include routines, programs,objects, components, data structures, etc. that have the technicaleffect of performing particular tasks or implement particular abstractdata types. Machine-executable instructions, associated data structures,and program modules represent examples of program codes for executingsteps of the method disclosed herein. The particular sequence of suchexecutable instructions or associated data structures represent examplesof corresponding acts for implementing the functions described in suchsteps.

Embodiments may be practiced in a networked environment using logicalconnections to one or more remote computers having processors. Logicalconnections may include a local area network (LAN) and a wide areanetwork (WAN) that are presented here by way of example and notlimitation. Such networking environments are commonplace in office-wideor enterprise-wide computer networks, intranets and the internet and mayuse a wide variety of different communication protocols. Those skilledin the art will appreciate that such network computing environments willtypically encompass many types of computer system configurations,including personal computers, hand-held devices, multiprocessor systems,microprocessor-based or programmable consumer electronics, network PCs,minicomputers, mainframe computers, and the like.

Embodiments may also be practiced in distributed computing environmentswhere tasks are performed by local and remote processing devices thatare linked (either by hardwired links, wireless links, or by acombination of hardwired or wireless links) through a communicationnetwork. In a distributed computing environment, program modules may belocated in both local and remote memory storage devices.

An exemplary system for implementing the overall or portions of theexemplary embodiments might include a general purpose computing devicein the form of a computer, including a processing unit, a system memory,and a system bus, that couples various system components including thesystem memory to the processing unit. The system memory may include readonly memory (ROM) and random access memory (RAM). The computer may alsoinclude a magnetic hard disk drive for reading from and writing to amagnetic hard disk, a magnetic disk drive for reading from or writing toa removable magnetic disk, and an optical disk drive for reading from orwriting to a removable optical disk such as a CD-ROM or other opticalmedia. The drives and their associated machine-readable media providenonvolatile storage of machine-executable instructions, data structures,program modules and other data for the computer.

Technical effects of the method disclosed in the embodiments includeincreasing the utility and performance of remote imaging systems forobject detection, tracking and identification. An imaging and trackingsystem that parses spatial and spectral imagery to a form a uniqueidentifier that references and describes an observed object is robust tochanges in factors associated with the image collection process. Becausethe unique identifier is based upon the intrinsic characteristics of theobject that are independent of the characteristics of the imagingsystem, the tracking system can correlate separate observances of anobject when historical information about the object life is initiallyunknown. In other words, the system may observe a previously unknownobject, assign a unique identifier according to the method of thepresent invention and then observe the object at a later time andcorrectly associate the observations. Additionally, the method of thepresent invention enables remote observation and searching of objectswithout requiring multiple source confirmation to verify the object'sidentity.

Referring now to FIG. 1, a system 10 for the formation of a singleunifying identifier for referencing an object 30 includes at least oneimaging device 14 configured to record a series of spectral and spatialimages of the object 30. A processor 16 is configured to process thespectral and spatial images of the object 30, and software in theprocessor 16 includes instructions to observe and associate keycharacteristics of the object 30 in each of the series of images. Aprocessor onboard the platform may perform a processing of the series ofimages or may instruct transmittal of the series of images to a remotelocation for processing by a second processor or processing system(collectively termed “a processor” 16 and shown as a geographicallyremote unit from the imaging device 14 in FIG. 1). Based upon theassociation of key characteristics of the object 30, the processor 16may assign a unique identifier to the object 30.

The imaging device 14 may include one or more cameras capable ofcapturing and recording image data at specific wavelengths across theelectromagnetic spectrum. Imaging devices for use in the presentinvention may capture imagery in wavelengths defined by the infrared,visible and ultraviolet bands. Additionally, the imaging device 14 maybe configured to collect imagery such that the operable bands of thecamera are combined to form a panchromatic image or divided to form amultispectral or hyperspectral datacube. In the case of a panchromaticimager, the imaging device 14 may form a series of panchromatic imageswhere each image is a record of the total intensity of radiation fallingonto each pixel of the image. The relationship between the pixels andtheir relative intensities form the spatial content (or spatial imagery)of the collected series of images.

Alternatively, the imaging device 14 may form a series of images whereeach image includes data corresponding to bands of specific wavelengths.For example, a general purpose camera may form three representations ofan image in the visible spectrum; one responsive to wavelengthsassociated with the color green, another for red and a third for blue.Additional bands may be provided to form images outside of the so-calledvisible spectrum. Spectral imagers with more numerous bands, finerspectral resolution or wider spectral coverage range may be calledmultispectral, hyperspectral or ultraspectral depending upon theparticular number and relative wavelengths of the imaged spectral bands.In this way, each pixel includes a spectral signature, or vector ofintensities that relate the spectral content (or spectral imagery) forthe collected series of images. While many imaging devices 14 arecontemplated for collecting spatial and spectral imagery, in onepreferred implementation, the imaging device 14 includes a staring arrayhyperspectral video camera. However, other known hyperspectral imagingdevices may include a combination staring array color/panchromaticcamera with a fast scanning spectral camera.

The imaging device 14 may, for example, be stationary on a mobileplatform 12 or movable on a stationary platform 112 (as shown in FIG.2), or any combination thereof. On a movable platform 12, as shown forexample in FIG. 1, the imaging device 14 may image an object of interest30 where the footprint of the imaged area 18, 20, 22, 24 moves, in part,as a consequence of the movement of the platform 12. The movement of theplatform may be an arc 28 traversed by the imaging device 14, a line 26traversed by imaging device 14, or any motion dictated by theoperability of the platform 12.

On a stationary platform 112, as shown for example in FIG. 2, theimaging device 114 may move by rotation in a single axis on the platform112 to track and image an object of interest 30. In this case, theimaged footprint 116, 118, 120 follows an arc 122 to image the object ofinterest 30. In most cases, the object of interest 30 would not followthe same arc 122 of the footprint, in which case the perspective of thefootprint will change. As well, the object of interest 30 may bestationary or mobile.

It will be apparent that relative motion between the imaging device 14or 114 and the imaged object of interest 30 will change the perspectivebetween the imaging device 14 or 114 and the object of interest 30.Consequently, the observed spectral reflectance of the object ofinterest 30 will vary, at least in part, as a function of the changingrelative perspective.

FIG. 3 is a flowchart showing a method of generating a spatial andspectral object model using the system described above in FIGS. 1 and 2.Initially at step 200, an imaging device 14 may acquire and track anobject of interest 30 by capturing imagery that is both spatially andspectrally resolved. Herein referred to as “hyperspectral images” toindicate the presence of both spatial and spectral content or images,the actual imagery may be collected with imaging devices as describedabove and may include elements responsive to multiple ultraviolet,infrared and/or visible wavelengths.

At step 202, an imaging device 14 may obtain a series of hyperspectralimages 203. To determine alignment among the hyperspectral images 203 inthe series, the processor 16 may employ image stability techniques toshift the series of hyperspectral images 203 from frame-to-frame tocounteract motion and jitter that may have been introduced, for example,by movement of the platform 12. The series of hyperspectral images 203of the object 30 may have a relative motion between the object ofinterest 30 and the imaging device 14.

At step 204, the processor 16 may determine at least one parameter 205of relative motion between the object of interest 30 and the imagingdevice 14. For example, the processor 16 may use data from an onboardsensor positioning system that measures relative and absolutepositioning. Example onboard systems may include relative positioningsystems like inertial navigation systems in combination with absolutepositioning systems like GPS. Along with the onboard positioning data,the processor 16 may ascertain differences in the series ofhyperspectral images 203 to infer motion of the object of interest 30and estimate a range from the imaging device 14 to the object ofinterest 30. The processor 16 may determine relative motion asrotational (i.e. roll, pitch, yaw) and translational (i.e. x, y, z)changes between the imaging device 14 and the object of interest 30. Theprocessor 16 may parameterize the relative motion with Euler angles anddirection vectors. Other parameterizations of the relative motionbetween the imaging device 14 and the object of interest 30 may applydepending upon the implementation. The processor 16 may map theparameter 205 of the relative motion between the object 30 and theimaging device 14 to determine an orientation 207 of the object 30 ineach hyperspectral image in the series at step 206.

Upon determination of an orientation 207 of the object of interest 30 ineach of the series of hyperspectral images 203, the processor 16 at step208 may identify spatial portions 209 of the object of interest 30 ineach of the series of hyperspectral images 203. Then, at step 210, theprocessor 16 may assign a spectral signature 211 to each spatial portion209 of the object of interest 30 in each of the series of hyperspectralimages 203. Based on the assignment of a spectral signature 211 to aspatial portion 209 of the object of interest 30, the processor 16 maygenerate, at step 212, a multi-dimensional spectral reflectance profile213. The dimensionality of the spectral reflectance profile 213 isdetermined by the orientation 207 of the object 30, the spatial portions209, and the spectral signatures 211 associated with the spatialportions 209. Therefore, the multi-dimensional spectral reflectanceprofile 213 may describe both the spectral reflectance signatures 211 ofan object of interest 30 and the spatial relationships among thespectral reflectance signatures 211 along with a spatial, orgeometrical, description of the object 30.

Once the multi-dimensional spectral reflectance profile 213 isgenerated, the processor 16 may classify the object of interest 30 inthe series of hyperspectral images 203. The multi-dimensional spectralreflectance profile 213 encodes a description of the spatial dimensionsand spectral textures of the object of interest 30. The processor 16 mayimplement additional processing techniques as described below todetermine the size and shape, along with texture characteristics, of thespatial portions 209 of the object of interest 30.

Upon completion of the method at step 214, the imaging device 14 mayreacquire the object of interest 30 in successive series ofhyperspectral images 203. The processor 16 may improve themulti-dimensional spectral reflectance profile 213 of the object basedupon the successive collections of hyperspectral imagery. While initialpasses may result in unobserved orientations of the object, successivepasses may begin to fill in the model of the multi-dimensional spectralreflectance profile 213 for the previously unobserved orientations.

Conversely, the processor 16 may improve the multi-dimensional spectralreflectance profile 213 for previously observed orientations. When theprocessor 16 reacquires an object at a previously observed orientation,the processor 16 may update a previously generated multi-dimensionalspectral reflectance profile 113 by weighting the spectral signature 111based upon the integration time of the hyperspectral image. For example,if a given spatial portion 209 for a given orientation 207 has beenpreviously observed for 0.1 seconds to determine a spectral signature211 and then an additional measurement is made for 0.2 seconds, thespectral signature 211 for the spatial portion 209 for the orientation207 in the multi-dimensional spectral reflectance profile 213 may beadjusted to weight the new measurement twice as heavily as the oldmeasurement.

To illustrate, FIG. 4 demonstrates the spatial portioning of an imagedvehicle for three different orientations 300, 302, 304. For a firstimaged side of the vehicle at orientation 300, the processor 16identifies four spatial portions 310, 312, 314, 316. For a second imagedside of the vehicle at orientation 302, the processor 16 identifies fourspatial portions 318, 320, 322, 324. For a third imaged side of thevehicle at orientation 304, the processor 16 identifies four spatialportions 326, 328, 330, 332. The processor 16 then assigns a spectralsignature based on the hyperspectral imagery to each of the spatialportions. In this example, there will be four distinct spectralsignatures for each of the three imaged orientations for a total of 12distinct spectral signatures. Therefore, with respect to the methodoutlined in FIG. 3, the illustration of FIG. 4 demonstrates amulti-dimensional spectral reflectance profile 213 that includes threeorientations, each with four spatial portions 209 and each spatialportion includes one corresponding spectral reflectance signature 211.

The processor 16 may further analyze the spatial and spectral imagery toidentify uniquely the object of interest. That is, the processor 16 mayanalyze a spatial/spectral characterization (such as themulti-dimensional spectral reflectance profile described above) toderive and associate key characteristics of the object 30 with a goal ofidentifying the individual instance of the object being imaged. In thisway, beyond merely recognizing the type of object, the processor 16 mayfingerprint the particular object.

FIG. 5 is a flowchart showing a method of identification from a spatialand spectral object model. Initially, the processor 16 obtains a seriesof images of an object. As described above, the series of images mayprovide spectral and spatial content and may come from one or moreimaging devices configured to capture one or more types of imagery. Eachimaging device may provide spectral images, spatial images or acombination of both spectral and spatial images. Delineated by thewavelength of the captured imagery, imaging devices that may provide theimagery include visible 412, infrared 414, ultraviolet 416 andhyperspectral 418.

The processor 16 may then observe key characteristics of the object 30in the series of images. The processor 16 may derive key characteristicsfrom the spatial imagery using conventional image processing techniquesknown in the field of computer vision. Typical techniques relate toconcepts of feature extraction, detection, image segmentation, parameterestimation, image registration and recognition.

Additionally or in tandem with the spatial analysis, the processor 16may derive key characteristics from the spectral imagery usingtechniques known for the analysis of spectral imagery. Spectral-basedprocessing algorithms have been developed to classify or group similarpixels; that is, pixels with similar spectral characteristics orsignatures. A number of hyperspectral search algorithms have beendeveloped and used in the processing of hyperspectral imagery for thepurpose of target detection. These hyperspectral search algorithms aretypically designed to exploit statistical characteristics of candidatetargets in the imagery and are typically built upon well-knownstatistical concepts. For example, Mahalanobis distance is a statisticalmeasure of similarity that has been applied to hyperspectral pixelsignatures. Mahalanobis distance measures a signature's similarity bytesting the signature against an average and standard deviation of aknown class of signatures.

Other known techniques include Spectral Angle Mapping (SAM), SpectralInformation Divergence (SID), Zero Mean Differential Area (ZMDA) andBhattacharyya Distance. SAM is a method for comparing a spectralsignature to a known signature by treating each spectra as vectors andcalculating the angle between the vectors. Because SAM uses only thevector direction and not the vector length, the method is insensitive tovariation in illumination. SID is a method for comparing a spectralsignature to a known signature by measuring the probabilisticdiscrepancy or divergence between the spectra. ZMDA normalizes thesignatures by their variance and computes their difference, whichcorresponds to the area between the two vectors. Bhattacharyya Distanceis similar to Mahalanobis Distance but is used to measure the distancebetween a set of spectral signatures against a known class ofsignatures.

By establishing a framework where spatial and spectral processingtechniques may be integrated, the processor 16 may observe keycharacteristics of the object at step 420. Key object characteristicsmay include spatially based attributes such as object shape 422, objectsize 424, object location 426, and object velocity 428. Key objectcharacteristics that incorporate spectral attributes may include objectcolor 430, object material composition 432 and texture. Thesecharacteristics may provide the type of information typically used in aclassification process. For example, the combination of thesecharacteristics may indicate the type of vehicle observed by the imagingsystem.

The processor 16 may infer other key characteristics with additionalprocessing including object appearance 434, identifying object labels436, object behavior 438 and object history 440. The object appearance434 includes the nuanced and potentially unique aspects of the surfaceof the object. For example, a dent on a car or an additional antennamounted to the roof of a vehicle may provide a specific identifyingfeature that the processor 16 may observe and detect with spatial andspectral processing techniques.

Identifying object labels 436 may provide an anthropogenic identifieranalogous to the object appearance and may include license plates,bumper stickers, tail markings, etc. The processor 16 may include, forexample, algorithms for optical character recognition to furtherdiscriminate object labels. The processor 16 may observe object behavior438 and detect and quantify aspects of the object associated with thelimitations of the object such as a car's turning radius oranthropogenic attributes such as the level of adherence to traffic laws.

The processor 16 may also infer an object's history 440 by correlatingmultiple observations of an object across time. The time scale that theprocessor 16 may have to correlate across, that is the duration oftemporal discontinuity, may range from a few seconds such as when anobserved object is temporarily obscured to a time scale on the order ofdays when an object such as a vehicle that infrequently routes through aviewing footprint of the remote imaging device. Therefore, the object'shistory may establish patterns in location, behavior over time andchanges in physical appearance.

By associating key characteristics 442, the processor 16 maydifferentiate an object from other types of objects when keycharacteristics such as an object's shape 422, size 424, location 426,velocity 428 and color 430 are used to isolate the object of interest.Object isolation and differentiation across a temporal discontinuity mayrequire identification of other key characteristics such as materialcomposition 432, object appearance 434, identifying labels 438 above toestablish the object's identity.

To facilitate the referencing of objects of interest for archival andretrieval, the processor 16 may assign a single unique identifier 444 toreference the object through its life cycle as observed by the remoteimaging system. The unique identifier 444 may encode key characteristicsassociated with the object; that is, the visual, spectral and behavioralcharacteristics along with the historical characteristics as describedabove.

A system may manage a single or multiple identifiers simultaneouslydepending upon the objects in the current view of the system. Keycharacteristics may be aggregated together to create a new identifier todescribe an object observed by multiple systems. Due in part to theuniqueness of the identifier (and its one-to-one mapping to the objectit references), the identifier may provide an index to keycharacteristics of the object observed by the remote imaging system. Acomputing system (embodied above as processor 16) may automaticallygenerate the identifier or an operator of the computing system maymanually generate the identifier. Subsequent to the creation of theidentifier, the identifier may provide a reference to the object foradding new key characteristics or retrieving known characteristics ofthe related object.

With the advent of video hyperspectral sensors, one system may gathersufficient information to identify uniquely the observed objects.Multiple systems may act independently where each system may gathersufficient information to uniquely identify the observed objects. In themultiple system modality, information may then be shared among thesystems to aggregate key characteristics. In this way, keycharacteristics observable by one system may combine with different keycharacteristics observed by a second system to enhance the overallidentifier for the object.

A single unifying identifier for an object of interest that incorporatesmany of the observable and inferred characteristics of the object mayfacilitate key operations fundamental to remote imaging systems. Forexample, when using multiple remote imaging systems, the uniqueidentifier may facilitate an object handoff between systems by allowingfor the efficient transfer of the description of the object betweensystems. Additionally, the identifier may be used to enable predictingwhen or where a handoff may need to occur.

One of the key elements of the system is the fusion of spectralinformation derived from visible, infrared, ultraviolet, andhyperspectral imagers with spatial characteristics derived from moretraditional image processing techniques to define an object uniquely.The remote sensing system must use hyperspectral video or a hybridimaging system capable of capturing a timely sequence of imagescontaining both spatial and spectral imagery to allow for a continuouscorrelation of spatial and spectral information.

This written description uses examples to disclose the invention,including the best mode, and also to enable any person skilled in theart to practice the invention, including making and using any devices orsystems and performing any incorporated methods. The patentable scope ofthe invention is defined by the claims, and may include other examplesthat occur to those skilled in the art. Such other examples are intendedto be within the scope of the claims if they have structural elementsthat do not differ from the literal language of the claims, or if theyinclude equivalent structural elements with insubstantial differencesfrom the literal languages of the claims.

What is claimed is:
 1. An object referencing method, comprising:acquiring, from a first image capture device having a first perspective,a first set of images including spectral image data and spatial imagedata; acquiring, from a second image capture device having a secondperspective, a second set of images including spectral image data andspatial image data; identifying, based at least in part on the spatialimage data, a first set of spatial portions, of an object, based on thefirst perspective and a second set of spatial portions, of the object,based on the second perspective; associating, based at least in part onthe spectral image data, spectral signatures with spatial portionsincluded in the first and second sets of spatial portions; andgenerating a profile for the object including the first set and secondsets spatial portions and associated spectral signatures.
 2. The methodof claim 1, wherein the identifying the first set of spatial portions,includes determining a relative motion parameter between the object andthe first image capture device.
 3. The method of claim 1, wherein theidentifying the first set of spatial portions, includes determining anobject orientation in the subset of the first set of images.
 4. Themethod of claim 1, wherein the generating the profile, includesgenerating a multi-dimensional spectral reflectance profile.
 5. Themethod of claim 1, wherein the acquiring the first set of images,includes acquiring hyperspectral video.
 6. The method of claim 1,wherein the acquiring the first set of images, includes acquiring atleast one of a multispectral or hyperspectral datacube.
 7. The method ofclaim 1, further comprising classifying the object based at least inpart on the profile.
 8. The method of claim 1, wherein the generatingthe profile includes associating, for the object, at least one of a setof spatial dimensions or a set of spectral textures with the profile.